…NVIDIA#15134)

Source-Commit: ec3da87a9b688016f8f76cc43be9cf30c92c868e
Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>

…NVIDIA#15134)

Source-Commit: ec3da87

Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>

…t_token_ids

In disaggregated serving the orchestrator is a single asyncio event loop that,
per request, handles the context server's prompt_token_ids (~40k ints for long
agentic prompts) three times on that one loop: json.loads of the ctx response,
pydantic-validate into ChatCompletionResponse, then model_dump_json re-serialize
onto the generation request. At high concurrency this is the binding host
bottleneck and sits on the TTFT critical path.

Add an opt-in fast path, config-gated via DisaggServerConfig.gen_tokids_ctxbytes
(default off, like gen_strip_message_history): the orchestrator instructs context
workers (via DisaggregatedParams.return_prompt_token_ids_b64 on the context
request) to serialize prompt_token_ids into the context response as a
base64-encoded int32 buffer (one JSON string). The orchestrator then relays that
string verbatim onto the generation request without materializing the int list on
its event loop; the generation worker decodes it back. The encode runs on the N
context workers instead of the one orchestrator loop. Off => byte identical to
baseline.

Measured (DeepSeek-V4-Pro 6p1d, GB300, AA-RWLT coding-agent):
  - c2048 (stable, GPU-bound): TTFT p50 -57% (2.95s->1.28s), p95 -41%.
  - c2600 (orchestrator-bound): +74% mean completions; raises the
    sustainable-concurrency-before-collapse ceiling.

Builds on NVIDIA#15366 (body-strip) and is the disagg-HTTP-wire analogue of NVIDIA#15134
(executor IPC int32 bytes). Safe for text-only, non-harmony deployments.

Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

…t_token_ids

In disaggregated serving the orchestrator is a single asyncio event loop that,
per request, handles the context server's prompt_token_ids (~40k ints for long
agentic prompts) three times on that one loop: json.loads of the ctx response,
pydantic-validate into ChatCompletionResponse, then model_dump_json re-serialize
onto the generation request. At high concurrency this is the binding host
bottleneck and sits on the TTFT critical path.

Add an opt-in fast path, config-gated via DisaggServerConfig.gen_tokids_ctxbytes
(default off, like gen_strip_message_history): the orchestrator instructs context
workers (via DisaggregatedParams.return_prompt_token_ids_b64 on the context
request) to serialize prompt_token_ids into the context response as a
base64-encoded int32 buffer (one JSON string). The orchestrator then relays that
string verbatim onto the generation request without materializing the int list on
its event loop; the generation worker decodes it back. The encode runs on the N
context workers instead of the one orchestrator loop. Off => byte identical to
baseline.

Measured (DeepSeek-V4-Pro 6p1d, GB300, AA-RWLT coding-agent):
  - c2048 (stable, GPU-bound): TTFT p50 -57% (2.95s->1.28s), p95 -41%.
  - c2600 (orchestrator-bound): +74% mean completions; raises the
    sustainable-concurrency-before-collapse ceiling.

Builds on NVIDIA#15366 (body-strip) and is the disagg-HTTP-wire analogue of NVIDIA#15134
(executor IPC int32 bytes). Safe for text-only, non-harmony deployments.

Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

…t_token_ids

In disaggregated serving the orchestrator is a single asyncio event loop that,
per request, handles the context server's prompt_token_ids (~40k ints for long
agentic prompts) three times on that one loop: json.loads of the ctx response,
pydantic-validate into ChatCompletionResponse, then model_dump_json re-serialize
onto the generation request. At high concurrency this is the binding host
bottleneck and sits on the TTFT critical path.

Add an opt-in fast path, config-gated via DisaggServerConfig.gen_tokids_ctxbytes
(default off, like gen_strip_message_history): the orchestrator instructs context
workers (via DisaggregatedParams.return_prompt_token_ids_b64 on the context
request) to serialize prompt_token_ids into the context response as a
base64-encoded int32 buffer (one JSON string). The orchestrator then relays that
string verbatim onto the generation request without materializing the int list on
its event loop; the generation worker decodes it back. The encode runs on the N
context workers instead of the one orchestrator loop. Off => byte identical to
baseline.

Measured (DeepSeek-V4-Pro 6p1d, GB300, AA-RWLT coding-agent):
  - c2048 (stable, GPU-bound): TTFT p50 -57% (2.95s->1.28s), p95 -41%.
  - c2600 (orchestrator-bound): +74% mean completions; raises the
    sustainable-concurrency-before-collapse ceiling.

Builds on NVIDIA#15366 (body-strip) and is the disagg-HTTP-wire analogue of NVIDIA#15134
(executor IPC int32 bytes). Safe for text-only, non-harmony deployments.

Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

…NVIDIA#15134)

Source-Commit: ec3da87

Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>